Hugepages are not necessarily a good idea for stacks: they mean that every stack takes 2MB of RAM, even though most programs confine themselves to a few k of stacks. This patch adds a new GC tweak to optionally disable hugepages for stacks (Xnohugepage_stacks=1), by passing MADV_NOHUGEPAGE to madvise.

On this program:

let n = Atomic.make 0
let () =
  let _ = List.init 100 (fun _ -> Thread.create (fun () -> Atomic.incr n; Unix.sleep 100) ()) in
  while Atomic.get n < 100 do Thread.yield () done;
  ()

the new flag makes the memory consumption go from 400 MB to 200 MB. (I think the remaining 2MB per thread is the glibc pthread stack).